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Abstract 

This paper describes a telerobotics display system, 
the Multi-mode Manipulator Display System (MMDS), 
that has applications for a variety of remotely 
controlled tasks. Designed primarily to assist 
astronauts with the control of space robotics systems, 
the MMDS has applications for ground control of 
space robotics as well as for toxic waste cleanup, 
undersea, remotely operated vehicles, and other 
environments which require remote operations. The 
MMDS has three modes: 1) Manipulator Position 
Display (MPD) mode, 2) Joint Angle Display (JAD) 
mode, and 3) Sensory Substitution (SS) mode. These 
three modes are discussed in the paper. 

1. Introduction 

Manual control of a remote manipulator can be a 
difficult task due, in part, to a lack of useful feedback 
to the operator on the position of the manipulator with 
respect to its desired position, destination, or target 
object to be manipulated. For example, to control 
many remote manipulator systems, including the space 
shuttle remote manipulator system (SRMS), the 
operator relies largely on visual feedback from direct 
views through windows and indirect views from 
cameras. However, the visual information can be 
insufficient in providing the operator with adequate 
cues, due to obstructions, poor viewing angles, camera 
failures, or problems with resolution or camera control. 
Our first mode, the Manipulator Position Display 
(MPD) mode addresses these problems. 

Another area of which poses concerns for the 
operator is avoiding undesired positions which cause 
joint limits or singularities. The operator may not get 
an indication that such a problem is developing until 
the problem has already occurred. For example, a 
warning light may indicate that a reach limit has 
occurred. Such indications do not always warn the 
operator ahead of time so that he/she can avoid the 
undesired position, or provide useful cues to tell the 
operator how to get out of the situation once it has 
occurred. Our second mode, the Joint Angle Display 
(JAD) mode was designed to help alleviate some of 
these control difficulties. 

A third area of interest is force feedback through 
sensory substitution. Force feedback has been shown 


to be preferable to non-force feedback in many 
teleoperation studies. 1 However, providing force 
reflection in the form of a force to the operator's arm 
and hand muscles can have its disadvantages. Systems 
that provide force feedback are often bulky 
master/slave manipulators that are impractical in many 
environments. Further, presenting force feedback to 
the operator’s hand or arm in the presence of even 
small time delays has been shown to create operator 
induced instabilities. The third mode of the MMDS is 
the Sensory Substitution (SS) mode and addresses 
these issues. 

At the time of the writing of this paper, the MPD 
mode has undergone testing and is further along in the 
development cycle than both the JAD mode which is in 
its initial development and the SS mode which is still 
in its design phase. 


2. Manipulator Position Display (MPD) Mode 

The MPD mode provides six degree of freedom 
hand controller positioning cues to the operator in a 
graphical format. This mode was designed to help 
alleviate the problems associated with poor visui 
feedback caused by obstructions, poor viewing angles, 
poor resolution, camera control, or camera failure. The 
MPD mode relies on six degree of freedom 
information obtained from manipulator sensors, such 
as joint position encoders, or, if available, a computer 
based vision system which can calculate current 
position relative to a target or desired position. The 
MPD’s algorithms perform the necessary calculations 
and provide the operator with "fly-from" or “fly-to” 
cues that alleviate from the operator the burden of 
calculating the appropriate system inputs. 2 

In order to operate effectively, the MPD mode 
requires knowledge of the current and desired (or 
target) positions. The current position of the 
manipulator arm can be obtained through real time 
position data from the system sensors (encoders or 
vision system) in six degrees of freedom. The desired 
position of the arm in six degrees of freedom can be 
entered into the MPD program if they are known 
aprioi, or can be obtained from vision system or 
telemetry data if such data are available. With this 
knowledge, the MPD mode can present the deviation 
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or error that exists in each degree of freedom to the 
operator in an easy to use format. The MPD mode not 
only has applications for the space manipulators, but 
also for other human-machine applications (aircraft, 
deep sea manipulators, toxic waste cleanup, etc.) which 
require the operator to control multi-degree of freedom 
systems under limited viewing conditions when the 
desired target points can be identified. 


2.1 MPD Mode Display for the SRMS - The 

Romtional/Translational Display (RTD) 

The display to be used by the operator in MPD 
mode can be tailored to the application area if 
necessary. As an example, we will examine the 
Rotational/Translational Display (RTD) which is the 
MPD mode display designed for astronauts who 
control the Space Shuttle Remote Manipulator System 
(SRMS). 

Figure 1 shows the format of the RTD. 3 The RTD 
separates the rotational and translational cues by 
depicting those cues through the motion of two 
separate objects. Two separate hand controllers are 
used by the astronauts to control the SRMS: 1) a 
Translational Hand Controller (THC) to control all 
translational motions, and 2) a Rotational Hand 
Controller (RHC) which is used to control all rotational 
motions. The RTD was designed so that one object on 
the display would correlate exclusively to the 
translational inputs on the THC, while the second 
object would correlate exclusively to rotational inputs 
on the RHC. This concept and the format of the 
display was developed by working directly with 
several astronauts. An explanation of how the RTD 
works and a summary of its features follows. 
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Fig. 1. Rotational/Translational Display (RTD) 
Format 


The line in the center with the three tick marks in 
Fig. 1 is stationary and acts as the reference line. The 
operator drives the translational cues using the square 
with the tick marks shown. Deviation in Z-translation 
(up an down motion) is depicted by the square being 
above or below the reference line, while Y-translation 
deviation (side to side motion) is shown by the square 
being to the left or right of the center of the reference 
line. For X-translation (in and out motion), the 
operator relies on the size of the square relative to the 
length of the reference line. 

For rotational cues the operator looks to the 
circular object shown in Fig. 1 . The position of the 
circle with respect to the reference line provides the 
rotational deviation information. If the circle is above 
or below the reference line, a deviation in pitch exists. 
A deviation in yaw is depicted by the circle being to 
the left or right of the center of the reference line. Roll 
cues are provided by the orientation of the extended 
line running through the center of the circle and the 
shorter line in the center of the circle. If those lines are 
tilted to the left or to the right, then a deviation in roll 
exists. 

On either side of the display are bar graphs which 
represent the closure rate, on the left, and the closure 
distance, on the right. These cues become useful when 
the tip of the manipulator is approaching its final 
destination. In addition, the operator is provided with a 
digital readout of the deviations in each of the six 
degrees of freedom. This digital readout can be seen in 
the upper left hand comer of Figure 1, and would be 
helpful in the final stages of a task to ensure that the 
deviations are within the desired limits (i.e. close to 
zero). 

The RTD also includes a number of other features 
to provide the operator better assessment of the 
manipulator’s position. One of these features is the 
highlighting of cues. This feature becomes most useful 
when the manipulator is reaching its target position and 
the RTD cues are converging on the stationary 
reference line. A task will usually have defined 
tolerance limits for each degree of freedom within 
which the manipulator is considered to be at its desired 
final position. Based on this information the 
highlighting feature indicates to the operator when the 
manipulator is within the defined limit for each degree 
of freedom. This indication is achieved by increasing 
the width of specific lines on the rotational and 
translational cues. For example, when the position of 
the manipulator is within the specified range in the X- 
axis the square becomes bolder than the other lines. 
When all of the lines which comprise the translational 
cue are bold, the operator will know that the 
manipulator tip is within tolerance in the X, Y, and Z 
axes. The rotational cues work similarly. For 
example, the circle becomes bold when the 
manipulator’s attitude is within the yaw limit. As with 
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the translational cue, when the manipulator attitude is 
within limit in yaw, pitch, and roll the entire rotational 
cue will be bold. Fig. 1 shows an example of the bold 
feature indicating that the X-axis and the yaw axes are 
within range. The tolerances can be set to different 
values for each degree of freedom and for each task. 

In addition to the highlighting feature, the RTD 
display also provides color cues to help distinguish 
between the translational and rotational cues, and the 
stationary reference line. The use of color is useful 
when the manipulator position is close to its final 
destination and it can be difficult to differentiate 
between the translational cue, rotational cue, and the 
reference line. In the current MPD implementation the 
translational cue is drawn in red, the rotational in green 
and the reference line in white. 

The RTD also displays hand controller Direction 
Cues which provide the operator with cues for the 
necessary hand controller deflections. The Direction 
Cues can be seen in Figure 1 as letters following the 
deltas in the upper left-hand comer of the display. The 
letters I or O are used to indicate in or out deflection of 
the translational hand controller, L or R for left or right 
deflection of the translational hand controller, and U or 
D for up or down deflection of the translational hand 
controller. For the rotational Direction Cues the letters 
U, D, L , and R are used in the same way as with the 
translational Direction Cues. With Direction Cues the 
operator is presented with clear indications of the 
necessary hand controller deflections eliminating the 
possibility of unnecessary and potentially dangerous 
movement of the manipulator. 

The RTD also provides the operator with a choice 
for displaying the cues in fly-from (outside-in) or fly-to 
(inside-out) formats. 4 At the beginning of each task 
the operator is given the choice of which convention to 
use. Once the selection is made, the RTD lists the 
selection being used in the top center part of the screen 
as shown in Figure 1. 

The RTD also has the capability to select between 
the different coordinate frames in which the 
manipulator position and attitude can be commanded. 
The RTD can be operated in three different coordinate 
frames: orbiter, end effector, and payload. These 
choices correspond to the reference frame options for 
commanding the SRMS on the space shuttle. Once 
selected the choice is displayed in the top center part of 
the main display screen above the fly-from fly-to 
selection (see Figure 1). The choice of coordinate 
frames can be modified to include any number of 
frames. 

To quantify the effectiveness of the RTD, 
experiments with human operators were conducted. 
The RTD was presented to four trained and 
experienced test subjects on a GRID 1660 laptop 


computer. A space shuttle SRMS task was simulated 
using the Manipulator Analysis - Graphic, Interactive, 
Kinematic (MAGIK)^ simulation system which runs 
on Silicon Graphics computers. The task was a space 
station assembly task, which focused on the installation 
of a Pressurized Mating Adapter (PMA) to a space 
station module. The experimental results concluded 
that using the RTD significantly improved operator 
performance by 33% over performing the same task 

without the use of the RTD.^ 

\ 

The RTD is a useful tool for SRMS operations. 
Changes can be made to the RTD to tailor an MPD 
mode display for applications other than SRMS such as 
undersea, rovers, or toxic waste cleanup. 


Another area of telerobotics operation where the 
operator can use assistance is the avoidance of 
unwanted joint positions such as joint limits or 
singularities. Reaching such limitations could shut 
down the system with a software stop (soft stop) or a 
mechanical hardware stop (hard stop). This type of 
situation could force the operator to control the 
manipulator in a single joint mode where each joint 
must be driven individually to alleviate the problem. 
The goal of the Joint Angle Display (JAD) is to present 
the operator with graphical cues which provide 
information on the current position of each joint 
relative to software stops and hardware stops. 

The JAD is comprised of a set of bar graphs which 
represent the position of each joint of a manipulator. 
The bar graphs are updated on a real-time basis using 
data from the position sensors at each joint. The JAD 
mode has three submodes: 1) nominal operations, 2) 
joint limits, and 3) single joint operations. 



The nominal operations submode display provides 
the current joint positions to the operator. As can be 
seen in Figure 2, each joint is identified at the top of 
each bar graph: SY = shoulder yaw, SP = shoulder 
pitch, EP = elbow pitch, WY = wrist yaw, WP = wrist 
pitch, WR = wrist roll. As the position of the joints 
change the bar graphs are Updated keeping the operator 
informed of the position of each joint. Used in 
conjunction with the MPD mode display, the JAD 
provides the operator with sufficient information to 
keep the manipulator from reaching unwanted joint 
positions while being driven to its final POR. 
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Figure 2. Nominal Operations Joint Angle Display 
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3.2 Joint Limits Display Submode 


impossible. During these operations, the Single Joint 
Operations submode will not only provide the operator 
with information on the current joint positions and 
joint limits, but will also provide the operator with 
operational cues. These cues will include the amount 
of deflection needed for each joint, and the joint 
sequence. The sequence is shown by highlighting the 
bar graph associated with the joint to be commanded 
while the desired position is indicated by a triangle on 
the left-hand side to the bar graph. Once the joint 
reaches the desired position its bar graph is displayed 
normally and the bar graph associated with the next 
joint in the sequence is highlighted. Figure 4 provides 
an example of the Single Joint Operations Submode 
display. In this example the display indicates that the 
Wrist Pitch joint should be moved to -86 degrees. 


The second submode of the JAD includes all the 
features of the first submode plus cues to indicate the 
location of the task specific joint limitations. These 
limits are specific to different tasks being performed by 
the manipulator and can be used to keep the operator 
from positioning the manipulator in undesired areas. 
As can be seen in Figure 3 the joint limits are indicated 
by the small triangles to the right of each bar graph. 
Further, when a joint limit is reached the pattern or 
color of the associated bar graph can also change as an 
added cue for the operator. This feature eliminates the 
burden on the operator to recall the limit of each 
individual joint when trying to identify which joint has 
reached its limit. In addition, this display can also emit 
an audible tone when any joint reaches a limit. By 
including the audible tone the operator will be notified 
of a joint limit error without having to constantly 
monitor each joint. 



Figure 3. Joint Limits in the Joint Angle Display 
3:3 Single Joint Operations Disnlav Snbmode 

Another application for the JAD mode will be 
single joint operations when the operator needs to drive 
the arm through a sequence of single joint movements. 
This operational scenario could occur during failure 
modes which make controlling all joints concurrently 
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To provide force feedback information to the 
operator, force reflection is the method used for most 
master-slave systems. Force reflection has had a long 
history of success. For example, Hill and Salisbury ^ 
found in their experiments that with force feedback 
task completion times were significantly shorter than 
without force feedback for peg-in-hole tasks. 
However, providing force reflection in the form of a 
force to the operator's arm and hand muscles can have 
its disadvantages. Systems that provide force feedback 
are often bulky master/slave manipulators that are 
impractical in many environments. Further, presenting 
force feedback to the operator's hand or arm in the 
presence of even small time delays has been shown to 
create operator induced instabilities. Ferrell^ 
suggested that the advantages of force sensitivity could 
be maintained in the presence of a time delay if the 
force feedback were substituted through the auditory or 
tactile modalities, and that a tactile display to the active 
hand might be especially compatible. The third mode 
of the MMDS is the Sensory Substitution (SS) mode 
and addresses these issues. 

Bach-y-Rita, Webster, Tompkins, and Crabb^ 
define sensory substitution as "the provision to the 
brain of information that is usually in one sensory 
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4. Sensory Substitution Mode 


532 



domain (for example visual information via the eyes 
and visual system) by means of the receptors, pathways 
and brain projection, integrative and interpretative 
areas of another sensory system, (for example visual 
information through the skin and somatosensory 
system). Some examples include sign language for the 
deaf, and Braille for the blind." Sensory substitution 
has been successfully used for many years in helping 
people who are fully or partially deficient in one or 
more of their sensory systems, for example, sensory 
aids for the blind or deaf. 10 

4.1 Tactile and Auditory Displays 

To provide the sensory substitution information, 
we are concentrating on tactile and auditory feedback 
devices for the operator. The tactile, in particular 
vibrotactile, and auditory modalities are of interest for 
several reasons. Such displays might be particularly 
useful for presenting force information because they 
provided non-reactive representations of force 
feedback. Non-reactive means sense modalities that do 
not induce operator movements like force reflection 
does when providing force information.** Such 
movements may be undesirable in certain situations, 
and can cause instabilities in the presence of a time 
delay. They are desirable for generic task information 
as well, because the auditory and vibrotactile 
modalities can present information while not placing 
any extra burden on the operator's visual system which 
is normally intently viewing the remote task 
environment via television monitor. 

Vibrotactile and auditory displays may also 
provide cost benefits by reducing the need for 
expensive bilateral force reflecting manipulators. 
Further, auditory and vibrotactile displays may also 
reduce the need for expensive or complicated visual 
systems. Massimino and Sheridan* showed that force 
feedback could decrease the need for visual feedback, 
since force feedback combined with low frame rate 
conditions (3 frames per second) provided performance 
that was comparable to performance under high frame 
rate conditions (30 frames per second) without force 
feedback. In addition, Bliss, Hill, and Wilber* 2 
concluded that the utility of tactile feedback increased 
under poor visual conditions, and provided highly 
useful information that required a relatively low 
bandwidth channel. Thus a potential benefit of 
vibrotactile or auditory feedback is a possible reduced 
need for high quality visual feedback which could lead 
to decreased cost of teleoperation. 

We are currently developing auditory and tactile 
displays to present manipulator force and position 
information to the operator. These displays will be 
incorporated into our multi-mode system to provide the 
operator with an integrated visual, auditory, and tactile 
feedback display system. 


5. Conclusions 

The MMDS can be expected to provide significant 
operational benefits that include providing the operator 
with useful manipulator position information when 
viewing conditions are constrained, assisting with 
recognizing and avoiding unwanted manipulator 
position, and providing force information under 
conditions which would normally make the 
presentation of such information impractical. The 
MMDS can also reduce operator workload, reduce 
training time, and assist the operator with performing 
unscheduled or unpracticed procedures. The MMDS 
has space based application for the space shuttle and 
the space station as well as for ground control of space 
based manipulators. It is a generic system which can 
be utilized for dual use application areas such 
environmental, hazardous waste, nuclear, and undersea 
remote manipulation environments. 
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